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Abstract 



This paper considers a formulation of a differential game with constrained dynamics, where 
one player selects the dynamics and the other selects the applicable cost. When the game is 
considered on a finite time horizon, its value satisfies an HJB equation with oblique Neumann 
boundary conditions. The first main result is uniqueness for viscosity solutions to this equation. 
This uniqueness is applied to obtain the second main result, which is a unique characteriza- 
tion of the value function for a corresponding infinite time problem. The motivation comes 
from problems associated with queueing networks, where the games appear in several contexts, 
including a robust approach to network modeling and optimization and risk-sensitive control. 

Key words: Differential games, viscosity solutions, the Skorokhod problem, projected dynamics, 
queueing networks 

1 Introduction and motivation 

Queueing and service networks are used as models in many important application areas, including 
telecommunication, manufacturing, computer networks and vehicle traffic. The control of such 
networks is difficult and important, and is one of the major challenges facing modern systems 
theory. 

For many problems of this sort it can be difficult to accurately estimate a system model. 
Service distributions may be complicated and correlated, arrival rates may vary over time, and 
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so on. In such a case one may wish to consider a robust approach to control design. The goal 
of the present paper is to prove foundational results for such an approach, and in particular to 
develop some analytic machinery for the characterization and construction of value functions for 
robust formulations of queueing network optimization problems. These results are also useful for 
analyzing a number of related problems, including risk-sensitive control of queues and design of 
ordinary (rather than robust) stabilizing controls. 

The formulation we use to design robust controls is an analogue of the H°° formulation for 
unconstrained linear and nonlinear systems. See, for example, Appendix B of [3]. We consider a 
dynamic model whose evolution can be affected by two controls. One control attempts to keep the 
system in a good operating region (e.g., bounded queue lengths). Such a control may influence the 
system through service or routing policies. The other control, analogous to the "disturbance" in 
H 00 control, determines the value of various system parameters, such as service and arrival rates. 
This control will attempt to degrade system performance, and the optimization problem is posed 
as a game. 

When formulating a game of this type one must specify the exact form of the dynamics as 
well as the cost structure. We have two goals in mind regarding the formulation that is used in 
this paper. The first is to formulate a class of network optimization problems for which explicit 
solutions may be possible. The second is to choose a cost that will allow one to prove an analogue 
of the small gain theorem from H°° control. The small gain theorem, which is a key result in 
the robust control literature, precisely quantifies the robustness of controls designed using robust 
criteria against model perturbations. With the first goal in mind, we use the Skorokhod Problem to 
define the system model. This is the simplest dynamical model which preserves the essential features 
of a controlled network, and in fact the Skorokhod Problem often arises when one considers a law 
of large numbers approximation to a more detailed (e.g., discrete state) description of a queueing 
network. A consequence of using this model is that the value function of the differential game 
satisfies a first order, nonlinear partial differential equation (PDE) with linear Neumann boundary 
conditions (see Section 2). With both the first and second goals in mind, we have chosen a cost 
function that is essentially a "weighted" version of the time till the system state reaches the origin. 
The qualitative properties of unconstrained (e.g., linear system) and constrained (e.g., Skorokhod 
Problem) models are fundamentally different, and the standard sort of cost structures used in H°° 
control do not yield an analogue of the small gain theorem. The cost we use appears to be the 
proper replacement in the setting of constrained processes for the familiar quadratic cost used for 
unconstrained processes. The robust properties that result from the use of this cost are discussed 
briefly in Section 5, and a full discussion and proof of an analogue of the small gain theorem may 
be found in [1]. 

Very little is known regarding robust approaches to the control of queues. To the authors' 
knowledge, the only other work of this type is due to Ball, Day and Kachroo [2], which is motivated 
primarily by problems of automobile traffic control. Although the methods of analysis and cost 
structure of [2] are different from ours, there are many similarities in the system models. 

As noted above, the value function for our differential game satisfies (in a viscosity sense) a 
first order nonlinear PDE with oblique linear derivative boundary conditions. For several reasons, 
including ease of implementation, we focus on costs that yield controls that are independent of 
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time. Our main interest is therefore in a stationary optimization problem. Just as in the case 
of unconstrained processes ([17]) this PDE has poor uniqueness properties (even in the viscosity 
sense). In order to uniquely characterize the value function among all the possible solutions to the 
PDE, we relate the value function for this problem to the solution of an associated time dependent 
problem. As shown in Section 4, under appropriate conditions one can construct a solution to the 
time dependent problem from the solution to the static problem, and conversely. When the viscosity 
characterization of the unique solution to the time dependent problem is properly interpreted, it 
provides a analytic characterization of the particular solution to the static PDE that equals the 
value function for the differential game. 

The organization of the paper is as follows. In the next section we discuss the conditions 
required for uniqueness of the time dependent partial differential equation, and state and prove 
the comparison principle. Section 3 introduces the Skorokhod Problem and defines a game whose 
upper value solves the PDE of Section 2. Various special cases of this game have the same value 
as the games from queueing network optimization problems that are our main motivation, and 
this relation is made precise in Section 5. In Section 4 we introduce the static game, indicate its 
connection to the time dependent game, and characterize its value function. We close in Section 
5 with examples. The analytic characterizations of Sections 2 and 4 include as special cases other 
problems besides robust control, and we illustrate this feature by including a problem concerned 
with existence of stabilizing controls for a network and an optimization problem that arises in 
risk-sensitive control. 

2 Formulation of a PDE 

As noted in the introduction, the aim of this paper is to establish uniqueness results for viscosity 
solutions to a class of PDE. These PDE characterize the value functions of certain differential games 
that appear in problems of robust and risk-sensitive control of queueing networks. It turns out that 
just as in analogous problems involving unconstrained processes, one cannot prove uniqueness 
results for these static (time independent) PDE in the standard viscosity sense. However, there 
are some interesting features of the class of problems we study which allow us to circumvent this 
difficulty. In particular, we can define a finite time problem for which there is uniqueness, and 
under an additional condition obtain a simple set of necessary and sufficient conditions that must 
be satisfied by the value function for the time independent game. 

We first define the class of time dependent PDE. To simplify the exposition we use assumptions 
that are much stronger than necessary and consider as candidate solutions only Lipschitz continuous 
functions. The intended use of these results is in the construction of solutions, and for the problems 
we have in mind the value functions will be Lipschitz continuous. 

The domain of interest is the set R^ = {x £ R d : Xi > 0, i = 1, d}, where d £ IN is fixed. 
For each i £ {1, ...,d} we associate with the face Fi = {x £ R^ : Xi = 0} of R^ a vector 7;. It is 
assumed that > 0, and we further assume (without loss) that = 1. There is also an index 
set {1, J}, which in the control problem will correspond to the different possible "pure" service 
assignments (cf. the examples in Section 5). 
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We next state the time dependent PDE and related boundary and terminal conditions. For 
notational simplicity we consider only the time interval [0, 1]. For a function V : [0, 1] X jR d — > IR 
with independent variables (t, x) £ [0, 1] X jR d , DV(t, x) denotes the gradient with respect to the 
spatial variables x, and Vt(t,x) denotes the partial derivative with respect to t. For reals a and b 
a A b denotes the smaller of a and b and a V b denotes the larger. Let 

H{p) = /\ J 3=1 HAP)- (2-1) 

The function H defined in the last display will be the Hamiltonian for the PDE on the interior of 
jR^J_. Boundary conditions will also be needed, and these turn out to be linear oblique derivative 
conditions defined in terms of the vectors 7;. Let e 4 - denote the unit vector in coordinate direction 
i for i = 1, d. 

We use the following regularity conditions. 

Condition 2.1 For each j £ {1,..., J} the function Hj : jR d — > IR is convex. The functions 
f : U d + -> IR and g : [0, 1] X U d + -> IR are bounded and continuous. 

Note that Condition 2.1 implies Hj(p) is finite for all j £ {1, J} and p £ IR d . 

For a point x £ IR^J_ let I(x) = {i : Xi = 0}. We can now define (in formal terms) the PDE of 
interest: 

m&x{-V t (t, x) - H(DV(t, x));V(t, x) - g(t, x)} = 0, t £ (0, 1), x £ (IR+)°, (2.2) 

{DV(t,x),ji) = 0, t £ (0, £ I(x),x £ dU d + (2.3) 

V(t,0) = if t £ [0,1], (2.4) 

V(l, x) = f{x) for x £ K+\{0}. (2.5) 

The maximization in equation (2.2) is due to the fact that in the corresponding game we allow the 
minimizing player to stop the game and pay a stopping cost of g. This stopping cost can depend 
on the time and state of the controlled process. 

Let a bounded Lipschitz continuous function w : [0, 1] X IR^_ — > IR be given. We say that w is 
a viscosity subsolution [resp., viscosity super-solution] to (2.2)-(2.5) if the following conditions hold. 



Let : (0, 1) X Kq_ — > IR be a continuously differentiable function, and let (s, y) £ (0, 1) X (jR+\{0} 
be a local maximum [resp., local minimum] of w — 9. Then 



max I [-0 t (s,y)-H(D0(s,y))]A mm [-(D9(s,y), 7i )] I ; W (s, y) - g(s, y) \ < (2.6) 



[resp., 



max | \[-9 t (s, y) - H(D9(s, y))] V max [-(D9(s, y), 7i )]j ; w(s, y) - g(s, y)| > 0], (2.7) 

w(i,0) = ifi£[0,l], (2.8) 
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w(l,x)<f(x) [resp., w(l,x) > f(x)] if x £ K+\{0}. (2.9) 
If w is both a viscosity subsolution and supersolution, then it is called a viscosity solution. 

Now of course we will need some conditions on the vectors 7; that appear in the oblique boundary 
conditions. We will use the following two conditions. The first condition is intimately connected 
with the uniqueness theory for the underlying controlled processes (cf. [7, 11] and Section 3). We 
refer to [11] for detailed information on when Condition 2.2 is satisfied. A set B C K d is called 
symmetric ifx£B=?—x£B. 

Condition 2.2 Let vectors £ {1, ...,d} that satisfy (7;, e 4 -) = 1 be given. There exists a com- 
pact, convex, and symmetric set B with £ B° , such that if z £ dB and if n is an outward normal 
to B at z, then for all i £ {1, d} 

(z, ei) > — 1 implies (7;, n) > and (z, e 8 ) < 1 implies (7;, n) < 0. 
Condition 2.3 The vectors ji, i £ {1, d} are linearly independent. 

Remark Condition 2.2 is equivalent to the main condition of [7, 11], where it is used to study 
the Skorokhod Problem that is related to these boundary conditions (cf. Section 3). As phrased 
in these references, the condition requires that if z £ dB and e 4 -)| < 1 then (ji,n) = for all 
outward normals n to B at z. Clearly Condition 2.2 implies this latter condition. To prove the 
reverse implication, it suffices to show that (z,ei) < implies (ji,n) < 0. We use the fact that a 
set which satisfies the condition of [11] is automatically invariant under the oblique projection 

Lx = x - (x, e t )j t 

[11, Section 2.5 of part I]. Thus y = Lz = z — (z, ei)ji £ B. Since n is an outward normal to B at 
z, this implies — (z, e 8 )(ra, 7;) = (y — z,n) < 0. Hence if (z, e 8 ) < then (7;, n) < 0, as required. 

As is well known, in order to prove a comparison principle for viscosity solutions we need a 
suitable family of test functions. Our construction of the test functions is similar to that of [8], 
though much simpler. The test functions are constructed from the set B. As a preliminary step, 
we note that if B satisfies Condition 2.2 then so does the convex set 

B = -{x : min{||a; — y\\ : y £ B} < 8} 

for small 6 > 0. However, this set always has a smooth boundary, in the sense that if h(z) is the 
set of outward unit normals to B at z £ dB, then h(z) is single valued, and if Z{ £ dB — > z £ dB, 
then with an abuse of notation h(zi) — > h(z). We can therefore assume without loss of generality 
that B has a smooth boundary, and we do so for the remainder of the paper. 

The test function is constructed so that its level sets are all multiples of B. This will force the 
gradient of the function to inherit certain properties of the outward normals of B. In addition, we 
will need the function to grow in a quadratic fashion. 
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Lemma 2.1 Assume Condition 2.2, and for c £ [0, oo) let 

E(x) = c x £ d{cB). 

Then there exist < m\ < M\ < oo such that if x / and y = ax £ dB for some a £ (0, oo), 
then D'E(x) = bn(y) for some b £ [mi, Mi] (where b can depend on x and n(y) is the outward 
normal to B at y). Define the continuously differ entiable function ^(x) = (S(s)) 2 . Then there exist 
< m < M < oo and a function b : M d — > [to, M] such that D^(x) = b(x)'E(x)n(y) , and such that 
m IM| 2 < £( x ) < M||a;|| 2 . 

Proof Since £ B° and B is bounded and convex, £ is well denned. The properties of £ and £ 
then follow directly from the definitions. ■ 

Note that £ has linear growth along each radial direction. If we extend the definition of n by 
letting n(x) = n(y) if x = ay, a > 0, and y £ dB, then the expression D^(x) = b{x)'E{x)n{x) is 
valid for all x £ III d . 

In addition to the test function, we also need the following. 

Lemma 2.2 Assume Conditions 2.2 and 2.3. Then there exists a continuously differ entiable func- 
tion ii : —7- [0, 1] such that 

(Dn(x),ji) < 

for all x £ and all i £ I(x), and such that sup^^d ||Z)/i(a;)|| < oo. 

Proof This lemma would be a special case of [8, Lemma 3.2], except that the lemma in [8] assumes 
a bounded domain. (Conditions (B.l), (B.4) and (B.5) of [8, Lemma 3.2] hold trivially in our setup, 
and Conditions (B.6) and (B.8) are equivalent in our setup to Conditions 2.3 and 2.2, respectively.) 
However, given any compact set K C IR+, the proof of [8, Lemma 3.2] directly implies the existence 
of a continuously differentiable function v : IR+ — > [0, oo) with the following properties: 

• v has compact support, 

• Given any x £ K, (Dv(x),ji) < for all i £ I(x), 

• Given any x £ IR^, (Dv(x),ji) < for all i £ I(x). 

We can guarantee that the range of v is contained in [0, 1] by multiplying v by an appropriate 
constant. Hence if Si £ M, i £ IN satisfy Si > and J2iZi &i ^ 1> then 

oo 

8 = 1 

satisfies all the conclusions of Lemma 2.2. ■ 

Theorem 2.3 Assume Conditions 2.1, 2.2, 2.3, and let u (resp., v) be a viscosity subsolution 
(resp., super-solution) to (2.2)-(2.5). Then u < v on [0, 1] X IR+. 
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Remark Since a viscosity solution is by definition both a sub and a supersolution, the comparison 
principle stated in Theorem 2.3 implies uniqueness of viscosity solutions. 



Proof The general layout of the proof follows the standard arguments [6], while the treatment of 
the boundary conditions uses ideas from the proofs of [8] and [9]. Fix r G (0, 1), and for 5 > and 
c > let 

(1 - r) 2 

U(t, x) = u(t, x) — 5— - cp(x), 

(1 - r) 2 

V(t, x) = v(t, x) + 5— - — h cp(x), 

where p is the function described in Lemma 2.2. To prove the theorem it suffices to show that for 
all 5 G (0, 1) there is cq G (0, oo) such that 

U(t,x) < V(t,x) (2.10) 

if c G (0, Co) and (t, x) G (r, 1] X Let L < oo denote the larger of the Lipschitz constants of u 
and v. Given 5 > we can find u G (0, 1) such that 

\H{p+vi) - H(p + v 2 )\ < 5/2 if ||p|| < L + 1 and ||ui|| V ||u 2 || < u. (2.11) 

Let M — su P^g]R d 11-^ II - ^ e then let cq — (8/4) A (u>/4M), and assume henceforth that 
c G (0,c ). 

We will use the usual argument by contradiction, and so assume that 

p = SU p [U(t,x)-V(t,x)]>0. (2.12) 

For e G (0, 1) let 

&{t, s, x, y) = U(t, x) - V(s, y) - -^(x - y) - j-(t - s) 2 . 

Owing to the boundedness of u and v and the non-negativity of S ^~_jjj , fi and £, $ is bounded 
from above. We also observe that $ tends to — oo as either s or f tends to r, at a rate that is 
independent of (x,y) G (M+) 2 . Let the parameter a G (0, 1) satisfy 

a < p A (5/4) A (u/4M), (2.13) 

where M is a constant associated with £ in Lemma 2.1. Since a > there exist (t a ,s a ,x a ,y a ) 
which satisfy 

$(t ai s ai x ai y a ) > sup s, x, y) - a > p - a > 0. 

{t,.s,x,y)E{{T,l]xTR<l )2 

We then define the function 

s, a;, y) = s, a;, y) - | [(t - ^) 2 + (s - s ff ) 2 + f (a: - av) + f (y - y CT )] . 
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Note that if [(t - t a ) 2 + (s - s a ) 2 + £(x - x a ) + £(y - y a )] > 2 then 

$<r(t,s,x,y) < ®(t,s,x,y) - a 
< $(t a ,s a ,x a ,y a ) 

We conclude that the supremum in sup^ s x y ) e (( T i] X ]R d ) 2 ^ (^ s ' 2/) ^ s achieved, and label the 
point at which it is attained (t, s,x,y). It follows from 3> CT (f, s,x,y) > that 



^-(i - s) 2 + -£(x - y) < sup (\u(t, x)\ + \v(s,y)\) < oo. 

j£ £ (t,s,x,y)e([o,i]xm<{) 2 



Since £(x) > m||a;|| 2 , this implies t — s — > and ai — y— 7-Oase— t-0. According to (2.8) and (2.9), 
u(t, x) < u(£, a;) for 

(M)eEi[[0,l]x{0}]u[{l}xMf]. 

We claim that for sufficiently small e > neither (t, x) nor (s, y) can be in S. Indeed, if this is 
not true then there exists a subsequence along which both (t, x) and (s, y) converge to the same 
point in S. Using the continuity of u and v, this would violate the strictly positive lower bound 
(p — a) on at (t, s, x, y), since the limit superior of 3> CT (t, s, x, y) along such a subsequence would 
be bounded above by zero. 



We now consider the mapping 
(t, x) —7- u(t, x) 



Since this function achieves its maximum at (t, x) (j£ £ and u is a subsolution, equation (2.6) must 
hold. Letting 

0(t, x) = -^{x - y) + ^-(t - s) 2 + S ^~_ T ^ + c/x(a;) + °-{t - t a ) 2 + ^{x - x a ), 
we compute 

t (t,x) = ^{t-s)-5^^ + a(t-t a ) 
D9(t, x) = —b(x — y)^(x — y)n(x — y) -\ — b(x — x^E^x — x a )n(x — x a ) + cDp(x). 

8 2 

Now consider i such that Xi = 0, i.e., i G I(x). Since y G R+, it follows that (a; — y, e 4 -) < 
for all such i, and thus by Condition 2.2 that (ji,n(x — y)) = (ji,n(z)) < for i G (where 
z = a (a: — y),a > 0, and z G dB). The same argument also shows that (ji,n(x — x a )) < for 
i G I(x). Since by Lemma 2.2 (7;, Dp(x)) < 0, it follows that (7;, D9(t, x)) < for all i G I(x). 

According to (2.6), it must be true that 

-0 t (t, x) - H(D0(i, x)) < (2.14) 
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and u(t,x) < g(t,x). An analogous argument which uses the fact that v is a supersolution shows 
that either 

-Cs(s,y)-H(DC(s,y))>0, (2.15) 

with 

Cs(s,y) = -(i-s)+S ^_~ T \ -a{s-s a ) 

£ (S — T) z 

D((s, y) = -b(x - y)E(x - y)n(x - y) + ^-b(y a - y)E(y a - y)n(y a - y) - cDp(y) 

£ Z 

or else v(s,y) > g(s,y). Suppose that in fact v(s,y) > g(s,y) occurs infinitely often as e — > 0. 
Since u(t, x) < g(t, x), g is continuous, and a: — y — > and t — s — > 0, this would imply along the 
subsequence that 

lim sup (t, s, x, y) < lim sup [m(£, x) — v (s, y)] 

< lim sup [g (i, x) - g(s,y)] 

< o. 

Since this contradicts the lower bound (t, s,x,y) > p — a > 0, we conclude that (2.14) and (2.15) 
hold for all sufficiently small e > 0. 

We now subtract (2.14) and (2.15), and use \t — t a \ V |s — s a \ < 2 and (2.13) to obtain 

H(D0(t, x)) - H{D({s, y)) > -0 t (i, x) + C,(s, y) > 25 - 4a > 5. (2.16) 

Recall that L < oo serves as a Lipschitz constant for u and v. Hence the characterization of (t, x) 
and (s,y) as maximizing and minimizing points implies \\D9(t, x) || V ||Z?C(s, y) || < L. Let 

p = —b(x — y)E(x — y)n(x — y) 

£ 

v\ = — b(x — x^E^x — x a )n(x — x a ) + cDp(x) 
v 2 = 7}b(y a - y)E(y a - y)n(y a - y) - cDp(y). 

Recall that £ = £ 2 , where £ and £ are defined in Lemma 2.1. Then ^(x — x a ) < 2 implies 
£(x — x a ) < 2, and therefore ||ffr(a; — s (7 )£(a; — x a )n(x — x a )\\ < <rM. In a similar fashion one can 
show |||-6(?/ (7 — y)£(y (7 — y)n(y a — y)\\ < <rM. Also, since c < uj/4M, where M is an upper bound 
on \\Dp(x)\\, \\cDfj,(x)\\ V \\cDp(y)\\ < cM < uj/4. Together with (2.13) and the fact that u < 1, 
these bounds imply 

< u 

< u 

< L + ||ui|| < L + 1. 

Hence (2.11) holds. Since this contradicts (2.16) we conclude that equation (2.12) cannot be true. 
This implies (2.10), which completes the proof. ■ 
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3 The Differential Game 



In this section we formulate a differential game whose value function is a solution to the PDE of the 
last section. This is done for two reasons. One is to show existence of solutions. The second is so 
that certain scaling properties of the solution may be verified. In particular, it will show that the 
solution is linear in radial directions. However, it should be noted that the game we use for these 
purposes is not the game that motivates our study. The results of this paper will be used elsewhere 
to construct and analyze optimal service policies in a queueing network, where the optimization 
problem will be formulated as a game. For any given PDE there are a variety of games whose 
value function will define a solution to the PDE. We formulate here what seems to be the simplest 
possible game that will suit our needs. Examples of games that are directly related to queueing 
network optimization problems can be found in Section 5. 

The game will be formulated in the Elliot-Kalton sense [13]. We first give an informal descrip- 
tion. In order to properly define the dynamics of the controlled process we must introduce the 
Skorokhod Problem, which is also known in some contexts as the reflection mapping. The mapping 
defined by the solution to the Skorokhod Problem, which will be referred to as the Skorokhod Map, 
will allow us to construct controlled, constrained processes that are consistent with the boundary 
conditions used in Section 2. Of course in the queueing problems that motivate the PDE of Section 
2 it is the constraints on the state space of the network that define the boundary conditions for 
the PDE, and not the other way around. We give here the simplest formulation of the SP that will 
cover our needs. For a more general formulation see [11]. 

Let 

C Md+ ([0, 1] : R d ) = G C([0, 1] : R d ) : </>(0) G <}, 

where C([0, 1] : R d ) is the usual space of continuous functions with the sup norm metric. Let a set 
of vectors that satisfy Condition 2.2 be given. For each point x on the boundary of R^ let 




J2 a ^ 

iei(x) 

where as in Section 2 I(x) = {i : X{ = 0}. The Skorokhod Map assigns to every path ip £ 
Cjpid ([0, 1] : R d ) a path cf) that starts at <^>(0) = ip(0), but is constrained to R^ as follows. If cf) is 

in the interior of R^ then the evolution of cf) mimics that of tjj, in that the increments of the two 
functions are the same until cf) hits the boundary of R^j_. When cf) is on the boundary a constraining 
"force" is applied to keep (f> in the domain, and this force can only be applied in one of the directions 
d((f>(t)), and only for t such that (f>(t) is on the boundary. The precise definition is as follows. For 
rj G C([0, 1] : R d ) and t G [0, 1] we let \rj\(t) denote the total variation of rj on [0,t] with respect to 
the Euclidean norm on R d . 

Definition 3.1 Let tjj £ C^d ([0, 1] : R d ) be given. Then (<j),rj) solves the SP for tjj (with respect 
to R^J_ and ji, i = 1, d) if (f)(0) = ip(0), and if for all t £ [0, 1] 

1. <j>(t) = il>(t) + r](t), 
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2. 4>{t) e G, 

3. \ri\(t) < oo, 

4- \v\{t) = f [0it] l {<t>{s)€d n*}d\v\{s), 

5. There exists a Borel measurable function 7 : [0, 1] — > IR+ such that d\i]\-almost everywhere 



j(t) £ d(cj)(t)), and such that 



f](t)= [ y(s)d\ri\(s). 



J[0,t] 



Note that r] changes only when (f> is on the boundary, and only in the directions d((f>). 

Under Condition 2.2 the Skorokhod Map is Lipschitz continuous on the subset of C^d ([0, 1] : 



completely— S condition ([19]). Within the setup of the present paper this condition is also necessary 
for the existence of solutions ([4], [11, Section 4 of part I]). Note that Condition 3.1 implies Condition 
2.3 of Section 2. 

Condition 3.1 Given k C {l,...,d} there exist b{ > 0,i £ k such that if 7 = J2ie*. biji> then 
(7, ei) > for all i £ k. 

We next define a constrained ordinary differential equation. As is proved under Conditions 2.2 
and 3.1 in [7], one can define a projection ir : M d — > that is consistent with the constraint 
directions {7;, i = 1, d}, in that ir(x) = x if x £ and if a; £" IR^. then 7r(a;) — x = ar, where 
a > and r £ c?(7r(a;)). Figure 1 illustrates the projection for a two dimensional problem. 



M d ) for which solutions exist [7, 11]. The following condition is sufficient for this subset to be 
all of Cjpid ([0, 1] : III d ). This condition, which is rather weak, is known in the literature as the 



2 




G 



* 7t(x) 



7C(X) 




Figure 1: The discrete projection 



With this projection given, we can now define for each point x £ dM,+ 
projected velocity ([5]) 



and each v £ M, d the 
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For details on why this limit is always well defined and further properties of the projected velocity 
we refer to [5, Section 3 and Lemma 3.8]. Let (3 : [0, 1] — > M> d have the property that each component 
of (3 is integrable over each interval [0, T], T < oo. Then the ODE of interest is given by 

j>(t)=7T(<f>(t),P(t))- (3-1) 

An absolutely continuous function cf) : [0, 1] — > IR+ is a solution to (3.1) if the equation is satisfied 
in an a.e. sense in t. By using the regularity properties of the associated Skorokhod Map, one can 
prove that all the standard qualitative properties (existence and uniqueness of solutions, stability 
with respect to perturbations, etc.) hold [7, 10]. In fact, because of the particularly simple nature 
of the right hand side (i.e., 7r(^>(£), (3(t)) rather than 7r(^>(£), b(<f)(t)) + (3(t)) for some function b), 
one can show that (f> solves (3.1) if and only if (f> is the image of ip(t) = f (3(s)ds + x under the 
Skorokhod Map, in which case all such issues become trivial. 

The ODE (3.1) will define the dynamics in the game we consider. In order to complete this 
formal description of the game, we must also describe a cost structure. Associated with each of the 
convex functions Hj,j = 1, J, is its Legendre transform, which is defined for each (3 £ M d by 

L,(fi) = sup [(^p)-H J (p)]. 
P eTR d 

As is well known ([20]), each function Lj : M d — > IR U {oo} is never equal to — oo, convex on the 
set {(3 : Lj((3) < oo}, and lower semicontinuous. The maximizing player in the game will select 
a measurable control (3(t), which will control the dynamics through (3.1). The minimizing player 
will select a measurable control k(t), which will take values in the set {1, J}. This control will 
not directly affect the dynamics, but instead chooses which of the costs Lk((3) will be applied to 
the maximizing player. If ever <f)(t) = then play is stopped and a stopping cost of zero is paid. In 
addition, the minimizing player will choose a control X(t) that takes values in {0, 1}. If t is the first 
time that X(t) = 1, then play is stopped and a stopping cost of g(t,cj)(t)) is incurred. If at time 1 
play has not been stopped either by the minimizing player or by hitting the origin, then a terminal 
cost of f ((f)(1)) is paid. Hence the cost function for controls ((3, k, A) and initial condition <f)(t) = x 
at time t is defined as 



C t (x, (3, k, A) = 



tActAI 



L k (s){fK S )) ds + 1 {r«7,r<l}6'(^, H T )) + 1 {t>1, ( 7>1}/(<? !, (1)) 



where a = inf{s : <f)(s) = 0} and r = inf{s : A(s) = 1}. 

Although this completes the description of the dynamics and the cost structure, it does not 
fully describe the game, since we have not indicated which player has the "information advantage," 
nor have we indicated how that will be modeled. For the game we are interested in it is the 
maximizing player that is given the advantage [13]. Hence we consider the corresponding upper 
game. (Interestingly, for the games that are directly related to queueing optimization problems the 
value for the upper and lower games often coincide, and both equal the value of this upper game.) 

Let 

J t = {£; : [t,l] —7- {1, J} : k is Borel measurable} 

and 

B t = {j3: [t, 1] -> M d : (3 is Borel measurable}. 
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A strategy for the maximizing player at time t is a map v : Jt Bt with the property that v is 
nonanticipating: if k\(t) = A^^) for a.e. t G [s,r], then ^[^1] (4) = ^[^2] (i) f° r a - e - t & [ s > r ]- Let 
At be the set of mappings from [t, 1] to {0, 1}. Owing to the nonanticipating nature of strategies 
it would make no difference if we allowed the strategy for the maximizing player to also depend 
on the minimizing player's stopping control, and so to simplify this dependence is omitted. Let T t 
denote the collection of all strategies for the maximizing player at time t. Then the value function 
for the game is defined by 

V(x,t)=sup inf C t (x, i/[k], k, A). (3.2) 

In this paper we consider only Lipschitz continuous solutions. We therefore assume a compat- 
ibility condition between the stopping cost and the terminal cost, which is that f(x) = g(l,x) for 
x G K+. This condition will be satisfied by the time dependent problems used to analyze the time 
independent problem in Section 4. 



Theorem 3.2 Assume Conditions 2.1, 2.2, and 3.1, and that f{x) = g(l,x). LetV(x,t) be defined 
by (3.2), and assume that V is Lipschitz continuous. Then V(x,t) is the unique viscosity solution 
to (2.2)-(2.5). 

As is the usual case, the proof of this theorem follows from a dynamic programming principle. 
The following theorem can be proved using standard arguments [14]. 



Theorem 3.3 Assume Conditions 2.1, 2.2 and 3.1, and that f(x) = g(l, x). Let V(t, x) be defined 
by (3.2), and assume that V is Lipschitz continuous. Let < t < 1 and consider any point 
x G K+\{0} for which V(t,x) < g(t,x). Then there exists 80 G (0, 1 — t) such that for any 
S G (0, S ) 



V{t,x) 

where (f> solves (3.1) with (f>(t) 



sup inf 



t+S 



L Hs) ({3(s))ds + V(t + 5 l( f>(t + 5)) 



Proof of Theorem 3.2. Properties (2.8) and (2.9) follow directly from the definition of V(t,x). 
Thus we need only consider conditions (2.6) and (2.7). 

We first show that (2.7) holds. Let 9 : (0,T) X IR+ — > IR be a continuously differentiate 
function, and let (s,y) G (0,T) X (lR+\{0}) be a local minimum of V — 9. We can assume without 
loss that V(s, y) = 9(s, y). If V(s, y) > g(s, y) then (2.7) is satisfied, and so we must only consider 
the case when V(s, y) < g(s, y). If (2.7) is not true, then there must be a > such that 

9 t (s,y) + H(D9(s,y))>a (3.3) 

and 

(D9(s,y), lt )>a (3.4) 
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for all i G I(y)- Let So satisfy the conclusion of Theorem 3.3 for (t, x) = (s, y). 
The definition of H and (3.3) imply 

0t(s,y) + A/ = i sup [(f3,D9(s,y))-L j (f3)]>a. 

f3eTR d 

For j G {1, J} choose fij G M d so that 

t {s, y) + D0{s, y)) - L0,)] > |. (3.5) 

These vectors are used to define a strategy (in T s ) for the maximizing player by setting 

v[k]{t) = (3 k(t) 

for any control k G J s . Let (f> denote the controlled dynamics when this strategy is used and the 
initial condition is (f>(s) = y. Since the mapping y — > I(y) is upper semicontinuous (i.e., given any 
z G K d , there is e > such that I(y) C I(z) for all y G K d satisfying \\y — z\\ < e), we can assume 
that I(cj)(r)) C I(y) for all r G [0, 5o]. Suppose for a given r G [0, So] that A;(r) = j. In this case (3j 
can be related to <j){r) in a variety of ways, depending on the value of I(cj)(r)). However, since in 
all cases I(cj)(r)) C I(y), the definition of the Skorokhod Problem implies that 



iel(y) 



for some numbers a 4 - > 0, i G I(y)- Hence by (3.4), 

9 t (s, y) + (D9(s, y)Jj) + ^ y), a i7i ) > 9 t (s, y) + (D9(s, y),^). 

iel(y) 

When combined with (3.5) and the smoothness of 9, the last display implies 

d9(r,<f>(r)) 



dr 



Lk(r)(Pk(r)) > T 



for almost all r G [0, S] if S G (0, So) is sufficiently small. Hence for sufficiently small S G (0, So) and 
any control k 

9(s + S, 4>{s + S)) - 9(s, y) - T + L k{r) {v[k]{r))dr > ^ (3.6) 



However, by Theorem 3.3 



V(s, y) > inf 



S + S 



L k{r) (f3(r))dr + V(s + 6,<t>(s + 6)) 



Hence there is a control k G J s such that 



V(s,y) > 



,+s aS 
L Kr) {P{r))dr + V{s + 5,<f>{s + 5)) - —. 



14 



Subtracting the last display from (3.6) and using V(s, y) = 9(s, y) gives 

V(s + S, 4>{s + 5)) - 9{s + S, 4>{s + 8))<-^ 

o 

for all sufficiently small 8 £ (0, So). This contradicts the fact that (s, y) is a local minimum of V — 9, 
and so it follows that (2.7) is valid. 

To complete the proof we must show that (2.6) holds. Let 9 : (0, T) X — > IR be continuously 
differentiable, and let (s,y) £ (0,T) X (lR+\{0}) be a local maximum of V — 9. Once again we 
can assume that V(s, y) = 9(s, y). Since the minimizing player can always stop the game it follows 
that 

V(s,y)-g(s,y)<0. 
Thus if (2.6) is not true then there is a > such that 

9 t (s,y) + H(D9(s,y))<-a (3.7) 

and 

(D9(s,y), 7i )<-a (3.8) 
for all i £ I(y)- Let So satisfy the conclusion of Theorem 3.3 for (t, x) = (s, y). 

Let j £ {1, J} be a minimizer in Aj =1 Hj(D6(s,y)). We define a control for the minimizing 
player by setting k(r) = j for all r £ [0,#o]. Since this control is constant, a strategy for the 
maximizing player is just an open loop control (3(r) (which can depend on x and j). According to 
(3.7), 

9 t (s,y) + (D9(s,y),P(r)) - L k{r) (P(r)) < -a (3.9) 

for almost all r £ [0, So]. 

Let e > be such that \\y — z\\ < e implies I(z) C I(y)- First assume that ||<^>(r) — y\\ > e for 
some r £ [0, 8]. Since Hj(a) < oo for all a £ M d , Lj grows superlinearly: 




Also, there is M < oo such that for all x £ IR+ and v £ M d 

\\ir(x,v)\\ < M(l + |H|). 

Elementary estimates using the last two displays show that if S £ (0, So) is sufficiently small then 
whenever (f> travels more than e away from y in time S, 

9(s + 5, 4>{s + 5)) - 9{s, y) - L k(r) {v[k]{r))dr < (3.10) 

Next consider the case ||<^>(r) — y\\ < e for all r £ [0,S], where S £ (0,So). Then for almost every 
such r there exist numbers a 4 - > 0, i £ I(y) such that 

4>{r) = (3{ r )+ a tlt- 
iei(y) 
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By choosing S £ (0, So) smaller if need be, the smoothness of 9, (3.9) and (3.8) imply 

~ L k{r)ih{r)) = t (r,cf>(r))+(D0(r,cf>(r)),^) 

+ ( D0 ( r ^( r ))^^) - L k(r){h(r)) 

iel(y) 

< -a/2. 

Hence for sufficiently small S £ (0, So) and all /?(•) (3.10) holds. Arguing as before, this estimate 
and Theorem 3.3 contradict the fact that (s, y) is a local maximum of V — 0. M 



4 A Simple Characterization of the Value of an Infinite Time 
Game 

In the previous sections we proved a uniqueness result and representation theorem for the PDE 
associated to a finite time game. In this section we consider an infinite time analogue of this game. 
The value function for this game is of prime interest, in that it will play a key role in characterizing 
optimal robust service policies for certain network problems. It is analogous to the cost potential 
or storage function in standard deterministic robust control. 

One can formally write down a PDE that should be satisfied by value functions of this sort 
simply by deleting the Vj term in the PDE of Section 2 and assuming that g is independent of t. 
However, it turns out (as is also the case in the usual robust control setting, see [17]) that such 
equations have poor uniqueness properties, even in the viscosity sense. Hence we take a different 
tack. Let V(x) denote the value for the infinite time problem, and let V(t,x) denote the value 
defined in the last section. We will use the fact that if t < 1 is fixed, then V(x) and V(t, x) coincide 
as functions of x for all a; in a neighborhood of the origin. Under conditions stronger than those 
of the last two sections, we will prove (for an appropriate choice of the stopping cost g) that V(x) 
is entirely characterized by V(t,x), and, in particular, by V(0,x). This will allow the fact that 
V(t, x) is the unique solution to a PDE to be brought into consideration, and ultimately lead to a 
simple analytic characterization ofV(x). 

We begin by defining the infinite time game. As in Section 2, the dynamics of the game are 
defined by the constrained ODE (3.1), except that solutions are now considered on the infinite time 
interval [0, oo). We wish to define a game where play is not stopped until the state of the process 
hits the origin. Even though there is no fixed time by which the origin should be hit, we consider 
only controls for the minimizing player that will force this to happen. In particular, and in contrast 
with the game of Section 3, there is no other way for the minimizing player to stop the game. 

The game which satisfies these requirements can be defined as follows. Let 

J = {£; : [0, oo) —7- {1, J} : k is Borel measurable} 

and 

B = {/3 : [0, oo) — > M d : [i is Borel measurable}, 
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and let T denotes the set of non-anticipating maps from J to B. We then define 



V(x) = sup inf Cq(x, u[k], k), (4.1) 

j/gT kEj:a<co 



where cb is defined by (3.1), 



Cq {x, f3, k) 



L k{t) (f3(t))dt 



a = mf{t : cb(t) = 0}, and where the infimum over the empty set is oo. 



One can easily verify by a time rescaling argument that V is linear in radial directions, which 
d 
+ 



means that for all x G Hi and all a > 



V(ax) = aV(x). 

In this section we use the following additional condition. 

Condition 4.1 Each function Lj,j G {1, J} takes the form Lj((3) — 1, where 

L-(3) = { ° i f xeC i 
J 1 oo otherwise, 

with each Cj a closed nonempty convex set. 

This condition is satisfied by Examples 1 and 2 of Section 5, but not by Example 3. At the 
present time it is not known if one can generalize Theorem 4.3 to cover Example 3 as well. The 
game that defines V(x) has a particularly simple interpretation when Condition 4.1 holds. The cost 
Lj((3) = oo for (3 (j£ Cj acts as a constraint on the maximizing player, forcing (3(t) G Cj whenever 
the minimizing player chooses k(t) = j. Given that this constraint is satisfied a.s. in t, the cost 
CQ(x,(3,k) is simply the time that it takes to hit the origin, given a starting position x. Hence 
the goal of the maximizing player is to delay this event for as long as possible. The minimizing 
player wishes for this time to be as small as possible, and attempts to achieve this by selecting the 
constraint on (3(t) appropriately. 

We have observed that V(x) represents the optimal time till the origin is reached. Let g(x, t) = 
(1 — t) and R = {x : V(x) < 1}. We claim that when the stopping cost g is used in the game of 
Section 3 the solution is given for t G [0, 1] by 

V ( ' > - \ (1-t) otherwise. ( > 

The precise statement is as follows. 

Theorem 4.1 Let g(t,x) = (1 — t) and f(x) = 0. Assume that V defined by (4-1) is finite and 
Lipschitz continuous, and that Conditions 2.1, 2.2, 3.1 and 4-1 are satisfied. Then V(t,x) defined 
by (4-2) is the unique viscosity solution to (2.2)-(2.5). 
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Proof We give the proof for t = 0. The general case uses the same argument. Fix x £ R° and let 
e > be given. In the definition of V(x), choose v such that 

inf C%(x,v[k],k)>V(x)-e. 

This means for all k £ J that a > V(x) —e. We adapt v in the obvious way for use in the definition 
of the finite time problem V(0, x). Because v is non-anticipating this adaptation is well defined. If 
the minimizing player in V(0, x) never uses the stopping control, then they pay at least V(x) — e 
if a < 1 and 1 if a > 1. If they do stop play at some time s £ [0,1], then the cost will be at 
least s + g(s, (f>(s)) = s + (1 — s) = 1. Since x £ R, the radial linearity of V(x) and the definition 
of R imply 1 > V(x), and so in all cases the cost is at least V(x) — e. Since e > is arbitrary, 
V(0,x) > V{x). 

Next let v be any strategy that appears in the definition of V(0, x). Given e > we must find 
a control (k, A) for the minimizing player such that 

C (x, v[k], k, A) < V(x) + e. (4.3) 

Since x £ R° we can find e > such that V(x) +e < 1. Let v be any non-anticipating extension of v 
to trajectories k £ J that are defined on the interval [0, oo). Choose k £ J in the definition oiV{x) 
such that a < +e, and by abusing notation let k also denote the corresponding restriction to 
[0, 1]. Note that a < + e implies a < 1. By choosing to never exercise the stopping control, 

the minimizing player obtains (4.3). Since e > is arbitrary we find that V(0,x) < V(x). 

Finally we must consider x (j£ R° . Since the minimizing player can always stop play immediately, 
V(0, x) < 1. On the other hand, the same argument as in the case x £ R° shows that V(0, x) > 1, 
and so V(0, x) = 1. ■ 

The definition of viscosity solutions in section 2 is phrased in terms of test functions. An 
alternative definition in terms of sub- and superdifferentials is standard (see [3, 15]). The following 
result from [3] provides a link between the two definitions. It is stated in [3, Lemma II. 1.7] for an 
open domain, but the proof of the following version is identical. 

Lemma 4.2 Let Q denote (0,1) X (lR+ \ {0}) and let w £ C(Q). Then p £ D + w(x) [resp., 

p £ D~w(x)] if and only if there exists 9 £ C 1 (i7) such that D9(x) = p and w — 9 has a local 
maximum [resp., minimum] at x. 



As a result, the following statement is equivalent to the definition of Section 2. A bounded Lipschitz 
continuous function w : [0, 1] X IR+ — > IR is a viscosity subsolution [resp., supersolution] to (2.2) 

yd 



(2.5) if the following conditions hold. If (a,p) £ D + V{s, y), s £ (0, 1), y £ Ml \ {0} then 



max | |j-a - H(p)] A mmj_-(p,j t )]j ; w(s, y) - g(s, y) j <0 (4.4) 
[resp., if (a,p) £ D~V(s,y),se (0, l),y £ U d + \ {0} then 

max | I [-a - H (p)] V max [-(p, j t )] j ;w(s,y) - g{s,y) \ > 0]. (4.5) 
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Moreover, (2.8) and (2.9) hold. 



The following analytic characterization of V follows from Theorem 4.1. Note that even when V 
is smooth the set of sub and superdifferentials is multivalued on 9M!|. In particular, if p G D + V(x) 
[resp., p G D~V(x)] and x G <9IR+, then p + J2iel(x) a i e i e D + V(x) [resp., p - J2iel(x) a i e i e 
D-V(x)] for all a t > 0, i G I{x). 

Theorem 4.3 Assume that V defined by (4-1) is finite and Lipschitz continuous, and that Condi- 
tions 2.1, 2.2, 3.1 and J h l are satisfied. Then V satisfies the following conditions: 

p G D~V(x), x G i?+\{0} => H(p) A min (p, 7,-) < 0, (4.6) 

iei(x) 

V(0) = 0, (4.7) 

p G D + V(x),x G i?+\{0}, anrf 6 G [0, 1] [H(6p) - (1 - 6)] V max(6p,7 8 ) > 0, (4.8) 

iei(x) 

Conversely, suppose that a Lipschitz continuous and radially linear function V : IR+ — > IR is given 
that satisfies conditions (4-6)-(4-8), and such that the level set R = {x : V(x) < 1} is bounded. 
Then V equals the value function defined by (4-1)- 



Proof Assume that the function V(x) defined by (4.1) is finite and Lipschitz continuous. Property 
(4.7) follows directly from the definition of V . Define V(t, x) by (4.2). Equations (4.6) and (4.8) will 
be proved by applying Theorems 3.2 and 4.1, and then using the alternative phrasing of conditions 

(2.6) -(2.9) in terms of sub and superdifferentials. 

First assume that x G [(1 — t)R]°, where t G (0, 1). For such points the definition of R implies 
that V(t,x) < g(t,x). If p G D~V(x) and x G M+\{0}, then by (4.2) (0,p) G D~V(t,x). Inserting 
this into equation (4.5) and using V(t, x) < g(t, x) gives (4.6). Next let x G d[(l — t)R] and t G (0, 1) 
be given. Then 

D+V(t,x) = {(-(1-6), ftp) :pG D + V(x),be [0,1]}. 
Together with (4.4), the last display gives (4.8). 

To prove the last sentence of the theorem, let V(x) be a Lipschitz continuous function and 
radially linear function that satisfies conditions (4.6)-(4.8). Define V(t,x) by (4.2). We will verify 
the sub and superdifferential forms of (2.6)-(2.9). Conditions (2.8) and (2.9) follow directly from 

(4.7) , the fact that f(x) = 0, and the definition of V(t,x) via (4.2). Next suppose that (a,p) G 
D+V(s,y), with s G (0,1) and y G K+\{0}. If y e ((1 - s)R)° then the definition of V(t,x) 
implies V(s,y) < g(s,y), a = 0, and p G D + V(y). In this case (4.8) with 6=1 implies (4.4). If 
y G d((l — s)R) then V(s, y) = g(s, y) and (a,p) = (—(1 — 6), bq) for some q G D + V(y), and again 

(4.8) implies (4.4). If y £ (1 - s)R then V(s,y) = g(s,y) and (a,p) = (-1,0), and (4.4) follows 
from the fact that under Condition 4.1 

H(0) = A/ =1 sup -Lj(/3) = 1. 

i3eM d 



19 



Lastly we turn to condition (4.5). Suppose that (a,p) £ D V(s,y), with s £ (0,1) and y £ 
K+\{0}. If y £ ((l-s)i?) then V(s,y) < g(s,y), and so a = and p £ D~V(y). In this case (4.6) 
implies (4.5). If y £" (1 — s)i? then y(s, y) = y(s, y), and so (4.5) holds as well. Having verified all 
of conditions (2.6)-(2.9), the proof of the theorem is complete. ■ 

5 Three Examples Prom Queueing Network Optimization 

In this section we present three examples. The aim is to show how the game defined in Section 3 
can be used to formulate and study problems in the optimal control of queueing networks. The 
examples all consider the same 4 class, 2 station queueing network, but with different cost criteria, 
motivations, and interpretations. Although we focus here on service control, the ideas can be 
applied to problems with routing control as well. 

We first describe the dynamical model that will be used for all three examples. The queueing 
system consists of four queues, labeled 1, 2, 3 and 4, and two servers, labeled A and B (see Figure 
2). Customers enter queue 1 at rate A £ [0,oo). Queues 1 and 4 (resp., 2 and 3) are served by 
server A (resp., B). Queue i is served at rate m when the respective server serves only this queue. 
The control in this problem is the choice of which class is served, and hence the control takes 4 
values. It will turn out that this control problem has the same value as the corresponding "relaxed" 
version, which allows that server to split service effort between customer classes. 



A B 

Figure 2: A system with four queues and two servers 

For particular fixed choices of the control policy, this system is studied for a deterministic 
model in [16] and for a stochastic model in [18]. It has been shown that this system can experience 
instability for non-idling service policies, even if the usual load conditions are satisfied. This non- 
trivial behavior with regard to stability criteria makes the example interesting from the point of 
view of control and robust stabilization. 

We next describe the Skorokhod Problem model that is appropriate for this system. Our 
"derivation" of the Skorokhod Problem is intended only as a heuristic. A more careful analysis can 
be applied to relate a variety of more detailed models to the model we use via a law of large numbers 
type approximation. At the level of modeling we consider (which is sometimes called the "fluid" 
level), the system is described by a continuous time, continuous state dynamical model. The state 
4> £ IR+ represents the queue lengths at sites 1, 2, 3 and 4. The role of the Skorokhod Problem is 
to apply the proper compensation to the state when a service is attempted at an empty queue. A 
key observation here is that this is the only role the Skorokhod Problem will play. In particular, 
the Skorokhod Problem will not be required to enforce non-idling behavior. In the setting of 



1 






4 
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multiclass problems, it is common when considering issues of performance analysis to restrict to a 
fixed non-idling service policy, which then must be reflected in the form of the Skorokhod Problem 
[12]. Unfortunately, this can lead to a Skorokhod Problem that is not well defined, in that there 
can be multiple solutions for a single input ip [18]. In the present paper we allow the service to 
be a control, which turns out to be extremely convenient in that it always leads to a well defined 
Skorokhod Problem. We will return to this point after the description of the Skorokhod Problem 
is complete. 

Since the state space has been identified, all that remains in defining the Skorokhod Problem 
is to identify the vectors 7;. To see how this can be accomplished, we first consider the evolution 
of the system away from any of the boundaries, i.e., at points where all queues are nonempty. The 
dynamics will of course depend on the current control. Let U = {(1,2), (1, 3), (4, 2), (4, 3)} denote 
the control space. Hence if the control u = (sa, sb) G U is chosen, the server A serves sa and server 
B serves sb- Suppose, for example, that u = (1,3). Then the dynamics away from the boundary 
with this choice of control are 

<j> = Aei + /ii(e 2 - ei) + /i 3 (e 4 - e 3 ) = (A - - /i 2 ,A*2, 0). 

The interior dynamics with the other choices of the control are obvious analogues. 

We next consider the behavior when one of the queues is empty. For i = 1,2, 3, 4, the constraint 
mechanism on the face {x G IR+ : I(x) = {i}} must compensate for any service that is attempted 
when the queue is empty. As noted previously we will not assume, a priori, that the controls are 
restricted to non-idling service policies. In fact, the controls that appear in the definition of the 
value functions are "open loop," which simply means that they are functions of time but not of 
the state of the system (see Section 3). It is certainly possible that some of these controls will 
allow parts of the system to idle even when there is work to be done. We will rely on optimality 
to indirectly eliminate such controls when they are not appropriate. Hence if control u = (1,3) is 
selected, and if queues 1 and 3 are empty, the servers continue to (attempt to) serve queues 1 and 
3, and ignore queues 2 and 4. Suppose that <j)\ is zero, and that all other queues are nonempty. 
Then regardless of the current state of the control, if a service were attempted to queue 1 then 
the compensating action is to increase queue 1 in proportion to the erroneous service, and to also 
reduce the length of queue 2 by the same amount. Thus the direction of constraint on this face is 
71 = e\ — e 2 . A similar argument applies at the faces which correspond to queues 2 and 3 being 
empty, and so 7 2 = e 2 — e 3 and 73 = e 3 — e^. Since the output of queue 4 is not fed into any other 
queue, 74 = e^. 

Those familiar with the Skorokhod Problem will recognize that we have obtained the same 
Skorokhod Problem as the one used to model a single class tandem queueing network with four 
stations. This means that by optimizing over the service policy and by using open loop controls, we 
effectively decouple the different customer classes, at least from the point of view of the Skorokhod 
Problem and related analytic machinery (e.g., the PDE of Sections 2 and 4). This significantly 
simplifies the analysis, and to the authors' knowledge this fact has not been observed previously. 
The situation is very much analogous to the classical theory of controlled unconstrained ODEs. 
Here the system might be modeled by the ODE (f> = b((f>,u), with b(-, •) a smooth function of its 
arguments and u the control. If one considers a fixed feedback control then u may not be smooth, 
and there can be difficulty in associating a well defined dynamical model. If one uses open loop 
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controls then there is no difficulty at all in defining quantities such as the system dynamics and 
the value function. Of course if one is primarily interested in feedback controls then in a sense the 
difficulty is only deferred, since one must eventually construct a feedback control that performs in 
at least a nearly optimal fashion. Nonetheless, the use of open loop controls (or open loop controls 
and strategies in the game case) is important, since quantities such as the value function can be 
properly defined and used to study the problem further. 

With the directions of constraint now identified, the system dynamics are now defined as in 
Section 3 through equation (3.1). Conditions 2.2 and 3.1 are easily verified for this model. See, for 
example, [7, 11]. All that remains is to specify the cost structure, and it is here that our examples 
differ. We begin with the simplest example. 

Example 1: Optimal Stabilizing Service Policy. We wish to define a problem in which the 
arrival and service rates have fixed values, and the issue is to design a control strategy such that for 
any given nonzero initial state, the process (f> is returned to the origin as quickly as possible. The 
index set {1, J} of the previous sections is now {(1, 2), (1, 3), (4, 2), (4, 3)}, corresponding to the 
4 "pure" service assignments. Here and for the examples to follow, it is easy to prove that the value 
functions and related PDE are the same as those one would obtain by allowing appropriate convex 
combinations of the controls, which can be interpreted as a splitting of effort between different 
queues. 

This problem can be included in the general model of Sections 2 and 4 via an appropriate choice 
of the L functions. The velocities away from all boundaries for the 4 controls are 

/5(i,2) = - Mi, Mi - M2, M2, 0), /3(i,3) = (A - Mi, Mi, ~M3, M3), 

/3(4,2) = (^, ~M2, M2, -M4), /?(4,3) = (^, 0, ~M3, M3 ~ ^i) ■ 

If u = (sa, sb), then we define 



Note that since the cost effectively constrains (3 once the minimizing control u is chosen, this 
problem is not actually a game. With this choice the total cost to be minimized is the time to hit 
the origin, and so V(x) gives the minimal time to reach 0, given the initial state x. Even this simple 
problem is interesting, since the existence of a finite solution to the PDE of Section 4 indicates 
the existence of a globally stabilizing policy, and in addition suggests a policy that is optimal with 
regard to the design criteria (returning a perturbed state to zero as quickly as possible). 

The PDE in this case takes a very simple form. We have 



Example 2: Robust Stabilization. We next consider an extension of the previous example. In 
this problem the rates A, ^1, . . .,^4 will not be fixed, and so the problem will be a genuine game. 





22 



We assume that the rates are restricted to intervals as follows: 



A G [a, A] = B , m G [m t , M t ] = B t , i = 1, . . . , 4. 

As in the previous example, away from the boundaries there is a well defined velocity associated 
with each choice of the rates. We wish to consider a game in which the maximizing player will 
choose the rates, subject to these constraints, in such a way as to delay hitting the origin for as 
long as possible. Such a game can be obtained by choosing the functions L u ((3) to be —1 if (3 is in 
the range of velocities allowed by the service assignment (sa,sb), and oo otherwise. For example 

, . J -1 if [i = (A - ^i, hi - fi 2 , ^2, 0) for some A G -Bo, Mi £ -Bi, H 2 £ B 2 , 
v ' ' I 00 otherwise 

Note that Condition 4.1 holds for these functions. 

The Hamiltonians associated with each assignment are the Legendre transforms of the L func- 
tions. We adopt the convention that the operations V and A preceed addition and subtraction, and 
succeed multiplication. Then 

H(i,2)( a ) = c + aa i V Aai + mi(«2 — ai) V Mi(«2 — «i) + m 2 (a3 — °i 2 ) V M 2 (a3 — a 2 ) 

H(i,3)( a ) = c + aa i V Aai + fii(«2 — Oi\) V Mi(a 2 — «i) + 1713(0x4 — a^) V M^(pn — a^) 
H(4,2)( a ) = c + a0i \ V Aai + 1712(0x3 — a 2 ) V M 2 (a3 — a 2 ) + ■m^ — a^) V M^ — a^) 

H(4,3)( a ) = c + a °i\ V Aai + ^3(^4 — 03) V Ms(«4 — 03) + TO4( — 04) V M4( — 04). 



Hence 



= A (sAjSs)e ^if (sAjSs) (a) 



c + aai V A«i + [mi(«2 — «i) V Mi(a 2 — ai)] A [m4(- 04) V M 4 ( — 04)] 
+ ["^2(«3 - a 2 ) v M 2(«3 - CU2)] A [m 3 (a 4 - a 3 ) V M 3 (a 4 - a 3 )]. 



The assumptions of Theorem 4.3 apply. Hence a function y satisfying (4.6)-(4.8) will provide 
the optimal time till the origin is reached. In particular, suppose one can identify a set of optimizing 
service controls. With these controls, the time till the origin is reached will be bounded by V(x) 
for any rates A and m that stay within Bq and Bi, i = 1, . . . , 4. A more detailed discussion of the 
sense in which such a service policy is optimally robust is given in [1]. 



Example 3: A Risk-sensitive Problem. As our final example we consider a problem in risk- 
sensitive control. Consider a set of nominal constant rates A, fii, . . . , Ji 4 and a parameter c > 0. For 
z G IR define 

, , _ J z log z — z + 1 if 2: > 

1 00 otherwise, 

with the convention OlogO = 0. As in the previous example, the rates of the system will be chosen 
by the maximizing player. In contrast to that example, the constraints will not be "hard", although 
there will be a penalty for any deviation from the nominal rates. The form of the penalty is the one 



23 



that would be associated with a risk-sensitive control problem, and this point is discussed below. 
For example, if the service assignment is (sa, sb) = (1, 3) then 

L(i,3){P) = -c + inf {X£(X/X) + ^(mi/mi) + ^3^3/^3) : /3 = (A - ^1,^1, -^3,^3)} . 

The obvious analogous definitions apply in the other cases. With h(a) defined to be the Legendre 
transform (e a — 1) of I, one can calculate 

H(a) = c + Xh(ai) + jlih(a2 — «i) A jl4h( — a^) + //2/i(«3 — 0^2) A //3/i(«4 — 03). 

The structure of the cost reflects a centering around the nominal system as measured via 
the function I. This function is significant from the point of view of large deviations theory, 
and the game structure allows for a link to related questions in stochastic risk sensitive control 
theory. Consider a stochastic model in which inter-arrival and service times are exponential random 
variables with rates X,fii, ■ ■ - ,fi4- Let the time be rescaled by the proportion n and the space by 
n~ l , and denote by a n the (random) time till the origin is reached starting at the fixed point x in 
the rescaled space. If a service control is applied so as to minimize the cost \ogEe ncUn , then the 
limit of n~ l log Ee nc(J " as n — > 00 is formally given by the value V(x) of the game above. As in 
Example 2, one can prove a robust optimality for a service policy that is designed in this way. This 
issue is discussed further in [1] 
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